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FORMULATING  THE  BROGDEN  CLASSIFICATION  FRAMEWORK  AS  A 
DISCRETE  CHOICE  MODEL 

EXECUTIVE  SUMMARY 


Research  Requirement: 

In  the  1950’s  Brogden  proposed  a  number  of  related  approaches  for  measuring  the 
classification  efficiency  of  a  predictor  battery  for  assigning  applicants  to  different  jobs  (Brogden, 
1954;  Brogden,  1955;  Brogden,  1959).  These  approaches  are  all  based  on  optimal  classification 
in  which  applicants  are  assigned  to  the  jobs  for  which  they  have  the  highest  predicted 
performance.  More  recent  implementations  of  Brogden’s  framework  have  been  developed  for 
computing  the  mean  predicted  performance  (MPP)  under  different  statistical  distributional 
assumptions  (De  Corte  2000;  Chen  and  Darby  1997).  These  implementations,  however,  do  not 
address  classification  policy  constraints  (e.g.,  cut  scores  and  gender  restriction),  applicant 
preferences,  or  the  impact  of  other  recruiting/classification  tools  available  to  the  Army  (e.g., 
monetary  incentives  that  channel  applicants  to  particular  job  training).  To  obtain  policy  guidance 
it  is  important  to  reformulate  Brogden’s  optimal  classification  framework  as  a  statistical  model 
that  can  inform  operational  classification  problems. 

A  discrete  choice  model  (DCM)  is  a  rich  modeling  framework  that  is  concerned  with 
agents  making  choices  among  alternatives.  In  the  Army  classification  situation,  the  discrete 
choice  process  can  describe  applicants  choosing  among  job  training  opportunities.  The  elements 
in  Brogden’s  work  are  comparable  to  those  in  a  DCM;  for  example,  assigning  an  applicant  to  the 
job  with  the  highest  predicted  performance  corresponds  to  an  individual  choosing  an  alternative 
with  maximum  utility.  The  goals  of  this  effort  are:  (a)  to  demonstrate  the  feasibility  of 
formulating  Brogden’s  optimal  classification  problem  as  a  DCM;  (b)  to  show  how  a  DCM  can 
accommodate  classification  policy  constraints,  such  as  cut  scores  and  gender  restrictions;  and  (c) 
to  explore  how  a  DCM  could  be  applied  in  an  expanded  classification  framework  that  takes  into 
account  applicant  preferences. 


Procedure: 

To  show  that  the  Brogden  classification  framework  can  be  formulated  using  a  DCM 
framework,  we  rescaled  the  augmented  criterion  estimates  proposed  by  Brogden  using  an 
arbitrarily  large  constant  and  then  added  independent  standard  Gumbel  errors  to  obtain  the 
mathematical  structure  of  utility  equations  in  a  DCM.  Using  this  transformation,  we  derived  a 
mixed  multinomial  logit  (MMNL)  classification  model  that  is  analytically  comparable  to  de 
Corte’s  (2000)  implementation  of  Brogden’s  model  based  on  an  assumed  multivariate  normal 
(MVN)  distribution  for  criterion  estimates.  We  also  derived  an  empirical  or  sample  based 
multinomial  logit  (MNL)  classification  model  that  can  accommodate  personnel  classification 
policy  constraints  and  is  robust  with  regard  to  the  distributional  assumption  for  criterion 
estimates.  We  illustrated  the  MNL  optimal  classification  model  by  evaluating  the  classification 


v 


efficiency  potential  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  and  the 
Tailored  Adaptive  Personality  Assessment  Screen  (TAPAS)  for  minimizing  6-month  attrition  in 
four  military  occupational  specialties  (MOS)  under  pure  classification  and  selection- 
classification  models  and  varying  classification  policy  requirements.  Lastly,  we  described  a 
hybrid  approach  for  evaluating  classification  potential  of  predictors  that  account  for  applicant 
preferences. 


Findings: 

Brogden’s  optimal  classification  framework  for  evaluating  potential  classification 
benefits  of  predictors  can  be  solved  using  DCM  methods.  The  population  based  MMNL 
classification  model  was  shown  to  be  equivalent  to  the  MVN-based  optimal  classification  model 
of  de  Corte.  The  empirical  or  sample  based  MNL  classification  model  provides  a  rich  framework 
that  can  accommodate  personnel  classification  policy  constraints,  such  as  cut  scores  and  gender 
restriction,  and  is  robust  to  the  functional  fonn  and  distribution  of  the  criterion  estimates.  In 
illustrative  examples,  while  changes  in  average  estimated  attrition  due  to  inclusion  of  policy 
constraints  were  negligible,  they  were  consistent  with  anticipated  effects  and  are  expected  to 
become  stronger  with  more  valid  criterion  models  and  greater  differentiation  in  eligibility 
constraints  among  jobs.  The  possibility  of  using  criterion  estimates  that  are  not  normally 
distributed  was  shown  to  have  practical  implications  for  the  analysis  results. 


Utilization  and  Dissemination  of  Findings: 

The  DCM  methods  developed  in  this  report  provide  alternative  approaches  for  evaluating 
potential  classification  benefits  of  predictors  that  are  numerically  convenient  and  can 
accommodate  classification  policy  constraints.  For  classification  problems  where  sufficient 
applicant  data  are  available,  the  empirical  MNL  classification  model  can  more  accurately 
evaluate  potential  classification  efficiency  compared  to  population-based  classification  models 
(i.e.,  de  Corte’s  approach  and  MMNL  classification  models). 
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FORMULATING  THE  BROGDEN  CLASSIFICATION  FRAMEWORK  AS  A  DISCRETE 
CHOICE  MODEL 
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FORMULATING  THE  BROGDEN  CLASSIFICATION  FRAMEWORK  AS  A 

DISCRETE  CHOICE  MODEL 


Research  Requirement 

In  the  1950’s  Brogden  proposed  a  number  of  related  approaches  for  measuring  the 
classification  efficiency  of  a  predictor  battery  for  assigning  applicants  to  different  jobs  (Brogden, 
1954;  Brogden,  1955;  Brogden,  1959).  These  approaches  are  all  based  on  optimal  classification 
in  which  applicants  are  assigned  to  the  jobs  for  which  they  have  the  highest  predicted 
perfonnance.  Earlier  efforts  by  Brogden  and  other  researchers  (e.g.,  Alley  &  Darby  1994;  Chen 
and  Darby  1997)  for  measuring  the  classification  efficiency  assumed  equal  predictor  validities, 
equal  predictor  inter-correlations,  and  equal  job  quotas.  Advances  in  mathematical  computing 
have  made  it  possible  to  apply  Brogden’ s  general  approach  for  measuring  classification 
efficiency  with  unequal  predictor  validities  and  inter-correlation  among  the  predictors  (e.g.,  De 
Corte  2000)  -  for  example,  to  evaluate  the  incremental  classification  efficiency  of  non-cognitive 
experimental  predictor  batteries  beyond  the  ASVAB. 

While  current  implementations  of  Brogden’ s  framework  may  be  sufficient  for  evaluating 
the  potential  incremental  classification  efficiency  of  competing  experimental  predictor  batteries, 
they  do  not  offer  guidance  for  formulating  operational  classification  policies.  These 
implementations  do  not  address  classification  policy  constraints  (e.g.,  cut  scores  and  gender 
restriction),  applicant  preferences,  or  the  impact  of  other  recruiting/classification  tools  available 
to  the  Anny  (e.g.,  monetary  incentives  that  channel  applicants  to  particular  job  training).  It  is  not 
surprising,  therefore,  that  realized  classification  efficiency  is  often  much  lower  than  levels 
suggested  by  optimal  MPP. 


To  obtain  policy  guidance  it  is  necessary  to  reformulate  Brogden’s  optimal  classification 
framework  as  a  statistical  model  that  can  inform  operational  classification  problems.  In  this 
regard,  a  promising  direction  is  to  fonnulate  Brogden’s  framework  as  a  discrete  choice  model 
(DCM).  Choice  analysis  is  concerned  with  how  agents  make  choices  among  alternatives,  where 
some  of  the  factors  are  observable  to  the  researcher  and  others  are  not  (and  are  treated 
stochastically).  In  the  Army  classification  situation,  the  discrete  choice  process  can  describe 
applicants  choosing  among  job  training  opportunities.  The  elements  in  Brogden’s  work  are 
comparable  to  those  in  a  DCM;  for  example,  assigning  an  applicant  to  the  job  with  the  highest 
predicted  performance  corresponds  to  an  individual  choosing  an  alternative  with  maximum 
utility.  Beyond  algorithmic  similarity,  a  DCM  would  provide  researchers  and  policy  makers 
(when  evaluating  or  formulating  classification  policy  tools)  with  a  statistical  framework  that  can 
accommodate  important  factors  in  real  world  classification. 

The  goals  of  this  effort  are:  (a)  to  demonstrate  the  feasibility  of  fonnulating  the  Brogden 
optimal  classification  framework  as  a  DCM;  (b)  to  show  how  a  DCM  can  accommodate 
classification  policy  constraints,  such  as  cut  scores  and  gender  restrictions;  and  (c)  to  explore 
how  a  DCM  could  be  applied  in  an  expanded  classification  framework  that  takes  into  account 
applicant  preferences  as  a  function  of  monetary  incentives  and  cognitive  /  non-cognitive 
predictors. 
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Background 


Br ogden  Classification  Framework 

Brogden  (1959)  proposed  to  measure  classification  efficiency  of  a  predictor  battery  using 
the  maximum  obtainable  average  predicted  performance  across  all  possible  ways  of  assigning  n 
applicants  to  m  jobs.  This  maximum  allocation  average  is  also  known  as  the  MPP  (mean 
predicted  performance)  and  the  allocation  that  produced  it  is  said  to  be  optimal. 

In  applications,  the  predicted  perfonnance  scores  of  a  given  applicant  for  the  m  jobs  are 
often  assumed  to  be  jointly  distributed  as  multivariate  normal  (MVN).  Under  the  special  case 
with  equal  predictor  validities,  equal  predictor  inter-correlations,  and  equal  quotas  across  jobs, 
optimal  classification  is  equivalent  to  assigning  applicants  to  the  jobs  corresponding  to  their 
highest  predicted  performance  scores.  Using  this  heuristic,  the  MPP  can  be  computed  by 
averaging  the  highest  among  the  m  predicted  performance  scores  of  applicants.  In  the  general 
case,  with  unequal  predictor  validities,  or  unequal  predictor  inter-correlations,  or  unequal  target 
quotas,  this  simple  strategy  does  not  work. 

Brogden  (1955)  proposed  an  assignment  strategy  based  on  the  augmented  predicted 
criterion  (or  augmented  criterion  estimate)  of  applicants,  in  which  job  specific  constants  are 
added  to  the  m  criterion  estimates  of  applicants.  Each  applicant  is  then  assigned  to  the  job 
corresponding  to  his  highest  augmented  criterion  estimate.  Brogden  showed  that  by  appropriately 
choosing  the  job  specific  constants,  assigning  applicants  to  jobs  in  this  manner  satisfies  the  target 
job  quotas  and  produces  the  maximum  average  predicted  performance  scores  (i.e.,  the  MPP). 

Applying  Brogden’ s  augmented  criterion  estimate  assignment  strategy  involves  non¬ 
trivial  analytic  and  numerical  computing  challenges.  Earlier  efforts  by  Brogden  and  other 
researchers  (e.g.,  Alley  &  Darby  1994;  Chen  and  Darby  1997)  for  measuring  the  classification 
efficiency  (i.e.  the  MPP)  of  a  predictor  battery  focused  on  the  special  case  with  equal  predictor 
validities,  equal  predictor  inter-correlations,  and  equal  quotas.  An  exception  is  De  Corte  (2000) 
who  developed  an  approach  for  the  general  optimal  classification  problem  by  analytically 
working  directly  with  the  multivariate  normal  joint  distribution  of  the  criterion  scores.  De 
Corte’ s  approach  will  be  used  in  this  research  to  represent  the  general  Brogden  framework  and  to 
examine  its  relationship  to  DCM  methodology.  Earlier  studies  used  de  Corte’s  approach  to 
evaluate  the  optimal  classification  of  experimental  predictors  in  the  Anny  (Ingerick,  Diaz,  & 
Putka,  2009). 


DCM  Job  Choice  Model  Applications 

In  contrast  to  classification  analysis,  in  which  individual  applicants  are  classified  to  jobs 
to  provide  maximum  benefit  for  the  organization  (i.e.,  maximize  MPP),  choice  modeling  focuses 
on  identifying  jobs  that  are  expected  to  provide  maximum  benefit  for  individual  applicants  (i.e., 
maximize  utility).  A  Job  Choice  Model  (JCM)  is  concerned  with  describing  actual  applicant  job 
choices  in  relation  to  their  characteristics  and  the  attributes  of  job  alternatives  under  a  random 
utility  maximization  (RUM)  assumption.  RUM  posits  that  among  all  available  alternatives,  an 
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individual  will  choose  the  alternative  that  has  maximum  value  to  him/her.  In  other  words,  a  JCM 
describes  optimal  jobs  for  an  applicant  while  a  classification  model  describes  optimal  jobs  from 
the  viewpoint  of  personnel  managers. 

Earlier  Army  personnel  classification  studies  used  various  forms  of  the  JCM  to  model 
applicant  MOS  choices  based  on  data  about  the  specific  military  occupational  specialty  (MOS) 
and  incentives  that  were  presented  to  applicants.  Diaz,  Ingerick,  and  Sticha  (2007a)  used  a  JCM 
based  on  a  nested  logit  model  to  simulate  applicant’s  choice  of  MOS  to  support  the 
implementation  of  an  unobtrusive,  simulation-based  evaluation  of  the  Enlisted  Personnel 
Allocation  System  (EPAS;  Sticha,  Diaz,  Greenston,  &  Me  White,  2007). 

In  a  later  project,  Diaz,  Ingerick,  and  Sticha  (2007b)  extended  the  model  to  consider 
prediction  of  MOS-tenn  of  service  (TOS)  combinations,  and  applied  to  analysis  of  the  impact  of 
increasing  the  individual  cap  on  recruiting  bonuses.  In  response  to  a  difficult  recruiting 
environment,  the  Anny  obtained  legislative  authority  to  increase  the  EB  program  from  $20K  to 
$40K.  The  increased  incentives  could  expand  the  recruiting  market  and  channel  applicants  from 
other  MOS  into  ones  with  higher  incentives.  The  main  focus  of  the  study  was  to  estimate  the 
channeling  effects  of  expanded  alternative  bonus  programs. 

More  recently,  Diaz,  Sticha,  Hogan,  Mackin,  and  Greenston  (2012)  estimated  a  JCM  that 
models  Anny  applicants’  MOS  and  TOS  enlistment  preferences  as  a  function  of  enlistment 
incentives  to  support  the  development  of  a  tool  that  can  assist  the  Enlistment  Incentive  Review 
Board  (EIRB)  allocate  incentives  to  the  MOS  and  TOS  options  that  provide  the  greatest 
incremental  benefit  to  the  Army.  They  implemented  the  analysis  capability  of  the  JCM  as  a 
proof-of-concept  Decision  Support  Tool  (DST)  that  allows  users  to  specify  incentive  policy 
scenarios,  predict  applicant  enlistments  by  MOS  and  TOS  and  associated  costs  for  each  policy 
scenario,  and  compare  the  results  across  different  policy  scenarios. 

While  their  goals  seemingly  compete  with  each  other,  the  maximization  underlying  the 
optimal  classification  model  and  the  JCM  are  comparable.  That  is,  assigning  an  applicant  to  the 
job  with  the  highest  predicted  perfonnance  or  criterion  estimate  corresponds  to  an  individual 
choosing  an  alternative  with  maximum  utility.  In  the  next  section,  we  will  transfonn  the  criterion 
estimate  to  obtain  the  same  mathematical  structure  as  the  utility  equations,  then  solve  the 
classification  problem  using  DCM  estimation  procedures. 
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DCM  Approximation  of  Br ogden  Classification  Analysis 

In  this  section  we  describe  how  the  traditional  Brogden  classification  problem  can  be 
formulated  using  a  DCM  framework.  We  will  begin  by  summarizing  De  Corte’s  implementation 
of  Brogden’s  augmented  criterion  estimate  assignment  strategy  and  then  propose  an  alternative 
approximate  implementation  that  borrows  ideas  from  DCM  methodology.  In  a  later  section,  we 
will  draw  insights  from  this  approximate  implementation  to  propose  an  approach  for  conducting 
classification  analysis  that  is  more  robust  with  regard  to  distributional  assumptions  and  can 
accommodate  important  policy  constraints  and  applicant  preferences. 

Multivariate  Normal  Implementation  of  Brogden  Optimal  Classification 

The  following  discussion  summarizes  De  Corte’s  implementation  of  Brogden’s  optimal 
classification  strategy.  Our  development  uses  a  slightly  different  approach  but  closely  follows 
the  main  ideas  in  his  work.  With  respect  to  notation  used  throughout  this  report,  we  will  use 
upper  case  letters  to  represent  random  variables  and  lower  case  letters  to  represent  observed 
(sample)  values;  random  vectors  and  random  values  will  be  presented  in  bold  font. 

Classification  Outcome  Vector 

Let  the  random  variable  X^  denote  the  estimated  or  predicted  criterion  score  of  the  /'th 
applicant  for  the yth  job.  Also  assume  that  the  vector  of  estimated  criterion  scores  for  a  randomly 
selected  applicant,  X.  =  is  distributed  as  multivariate  normal.  De  Corte  developed 

an  approach  for  the  general  optimal  classification  problem  based  on  Brogden’s  augmented 
criterion  estimate  assignment  algorithm.  This  algorithm  modifies  the  yth  criterion  estimate  for 
classification  purposes  by  adding  a  job-specific  constant  -A  to  obtain  the  augmented  criterion 
estimate  V,  lT)Ej.  =  A,  +  Xi; .  The  algorithm  assigns  the  /th  applicant  to  the  Ath  job  if 
Xi  c;  !ik  >  Xi  (.y:j  for  all  other  jobs  /  #  k  (i.e.,  X,. is  the  maximum  modified  assignment 
criterion).  Brogden  showed  that  given  suitably  chosen  job  specific  constants  >  >1 .  s),  applying  the 
algorithm  to  a  large  pool  of  applicants  will  approximately  produce  the  desired  job  allocation 
quota. 


Brogden’s  optimal  classification  assignment  strategy  can  be  viewed  as  mapping  from  the 

/H-dimensional  random  vector  of  criterion  estimates,  iXn . XfJ,  to  the  2-dimensional  random 

vector  ,  where  is  a  random  variable  that  takes  its  value  from  the  criterion 

estimate  for  the  optimal  job  (i.e.,  job  with  the  highest  augmented  criterion  estimate)  of  the  /th 
'  '  ' 

applicant,  and  A'  is  a  random  variable  whose  value  is  the  index  of  the  optimal  job.  Note  that 

(Xf  ,K.  )  completely  describes  the  outcome  of  the  algorithm  for  the  /th  applicant.  The  joint 

probability  density  function  (PDF)  of  this  outcome  vector  can  be  expressed  as  the  product  of  the 
probability  of  observing  the  criterion  estimate  x  for  the  kth  job  and  the  conditional  probability 
that  this  observed  score  is  the  highest  among  all  in  augmented  criterion  estimates.  That  is, 
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f(K™  =  k.X™  =  x)  =  p(A'Ch)  =  k,Xik  =  x) 

=  fXk(x)p(hf}  =  k\xik  =  x) 

=  fxkMKxU  +Aj<x  +  =  x.k*  j) 


The  two  probability  factors  in  the  last  line  above  were  completely  specified  by  De  Corte. 
The  first  factor  is  simply  the  univariate  normal  PDF  corresponding  to  the  Ath  criterion  estimate 
evaluated  at  x.  To  evaluate  the  second  factor,  De  Corte  used  the  lower  tail  probability  of  the 
( m  —  1  >  -dimensional  normal  distribution  obtained  by  conditioning  the  Ath  criterion  estimate  to 
v.1  For  discussion  purposes  we  will  leave  the  joint  probability  in  the  above  final  fonn. 

Job  Quota  Constraints 

We  now  obtain  two  key  components  in  De  Corte’s  approach.  The  first  component  is  the 
system  of  nonlinear  equations  representing  job  quota  constraints.  These  equations  equate  the  job 
quotas  to  the  percentages  of  applicants  that  the  augmented  criterion  estimate  algorithm  assigns  to 

(*) 

the  m  jobs.  These  percentages  are  just  the  marginal  probabilities  of  A".  that  can  be  obtained  by 

integrating  the  joint  PDF  of  the  outcome  vector  over  all  possible  values  of  X..  .  Thus,  the  m  job 

quota  equation  constraints  are 

f+OO 

I  fxik(x)p(xu  +  Aj  <x  +  Ak\xik  =  x>k  *j)dx  =  qk  ,k  =  1,  ...,m  (1) 

—  oo 

where  the  left-hand  side  integral  equals  the  marginal  PDF  /  m  ( k ).  Note  that  the  variables  in  the 
above  system  of  nonlinear  equations  are  the  unknown  job-specific  constants. 

Calculating  MPP 

A  second  component  in  De  Corte’s  approach  is  the  marginal  PDF  of  the  predicted 
criterion  scores  of  applicants  on  their  optimal  jobs  (i.e.,  the  X,  s).  We  need  this  PDF  to  evaluate 

i  ’■  i  i  '•  i 

the  MPP.  It  can  be  derived  by  summing  the  joint  PDF  of  the  outcome  vector  ( X.  ,K.  1  over  all 

Cfo) 

possible  values  of  A'  .  Conceptually,  this  means  anyone  of  the  m  jobs  can  be  the  optimal  job. 

Thus  the  marginal  PDF  of  AT  evaluated  at  x  is 


1  Note  that  in  De  Corte’s  derivation  x  is  the  observed  value  of  the  augmented  criterion  score,  while  in  our 
presentation  above  x  is  the  observed  value  of  the  unmodified  criterion  score. 
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vN> 

fzm(x)  -  £  W  =  *•*,“  =  *) 

fe=I 

m 

=  £  +  -»,<*  +  4»  |X»  =  x.k+  i) 

k=l 

Note  that  this  marginal  PDF  involves  job  specific  constants  -'Fas  unknown  parameters.  To 
completely  define  this  PDF,  we  solve  the  job  quota  constraint  system  of  nonlinear  equations  for 
the  -d.s. 


Having  defined  the  marginal  PDF  of  the  predicted  criterion  scores  of  applicants  on  their 

(ft) 

optimal  jobs,  the  MPP  may  now  be  evaluated  by  taking  the  expectation  of  X. 


-  +co 


MPP  =  E(x<ih))  =  2^  |  i/I]kW^  +Aj<x+  Ak \xtk  =  x,k±  j)dx 

k-1 

It  is  also  informative  to  express  the  MPP  as  the  weighted  average  of  job  —  level  MPPs 

r+co 

MPP  =1  x  y  f(x-k)  =  k.X™  =  x]  dx 

k  -oa  k=± 

v  r-  /W** =*) 

=Lq-Lx - f 


k- 1 


F  —co 
•  +CO 


<7*  f  x/(^£Cft)  =  *  =  kjdx 

.  —  CO 

fe-1 

= = *) 


fe=l 


(2) 


W£  °btained  the  thW  lme  b*  n0“n8  *hat  V  «  -  *  and  *he  defml,10“  of  a 

conditional  PDF.  The  conditional  average  performance  or  conditional  MPP,  E  |  X.  K.  =  k  J, 
is  useful  for  assessing  classification  efficiency  pattern  across  jobs,  while  the  conditional  PDF 
/ 1  -V.  "  =  v  K.  "  =  k  I  is  useful  for  describing  the  distribution  of  optimal  predicted  performance 
scores  of  applicants  in  the  Mi  job. 
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MMNL  Approximation  of  Brogden  Classification  Framework 

We  next  propose  an  alternative  approach  to  the  general  optimal  classification  problem  by 
borrowing  ideas  from  DCM  methodology.  This  alternative  approach  is  presented  below  as  an 
approximate  implementation  of  Brogden’s  optimal  classification  framework.  In  the  next  section, 
we  will  draw  insights  from  the  analytic  results  derived  below  to  propose  a  more  robust  and 
flexible  approach  for  conducting  classification  efficiency  analysis. 

Modified  Augmented  Criterion  Estimate 

We  begin  by  proposing  the  modified  augmented  criterion  estimate  for  the yth  job  of  the 
/th  applicant  given  by 

=  +  Xij)  +  Eij 


where  the  =Ws  are  independent  standard  Gumbel  random  variables,  A  is  an  arbitrary  scaling 
parameter,  and  (as  before)  -V,  ,s  are  the  criterion  estimates  and  -d  .s  are  job-specific  constants. 
Note  that  A',  is  simply  a  rescaled  version  of  the  original  augmented  criterion  estimate,  X, ;a)y  , 
with  an  additional  Gumbel  random  variable  term.  In  general  the  rank  ordering  of  A', . . ,s  can 
differ  from  that  of  A'^-.s  and,  therefore,  can  lead  to  a  different  optimal  job  for  the  /th  applicant. 
However,  we  can  make  A  suitably  large  so  that  =  «(  ,  predominantly  detennines 

the  rank  ordering  of  A', . ...  ,s.  Since  scaling  does  not  alter  the  rank  ordering  of  jobs,  a  modified 
optimal  classification  algorithm  based  on  A',  a,;,.  ,s  can  approximate  Brogden’s  optimal 
classification  algorithm  to  a  desired  degree  of  accuracy  (i.e.,  both  will  produce  the  same 
applicant-job  assignments)  by  choosing  an  arbitrarily  large  value  of  A .  The  idea  behind  this 
modified  augmented  criterion  estimate  approximation  is  based  on  McFadden  and  Train  (2000). 


We  make  the  following  observation  before  deriving  job  quota  constraint  equations  and 
the  PDF  of  the  optimal  criterion  estimate  of  applicants  under  the  modified  classification 
algorithm.  Given  fixed  criterion  estimates,  Xf  =  jq,  the  form  of  the  modified  augmented 
criterion  estimates  Aj  /  ^ ,  s  follow  that  of  the  utility  equations  of  a  DCM  with  an  “observed” 

linear  component,  represented  by  Ai  Aj  +  Xi  ; ),  and  an  “unobserved”  component,  represented  by 
the  Gumbel  random  variable  Etj .  This  means  that  the  conditional  probability  of  a  specific  job 
having  the  highest  modified  augmented  criterion  estimate  given  Xt  =  jq  has  the  form  of  the 
multinomial  logit  (MNL)  probability 


Pi.X f  =xJXm 


expUU,  -x.,)) 

£7=1  exp  (a(a, +  *(,)) 


Job  Quota  Constraints 
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Under  the  modified  optimal  classification  algorithm,  it  can  be  shown  (see  Appendix  A) 
that  the  job  quota  constraints  can  be  represented  by  the  system  of  nonlinear  equations  below, 
with  variables  -As: 


exp  ( A(A.  +  3£e>)) 

i**m  exP  WA  +  *«)) 


fx(Xi)dXi  =  qk,k  =  1 


(3) 


The  system  of  equations  (3)  corresponds  to  (1)  in  the  MVN  model.  Note  that  the  left- 
hand  side  of  each  equation  above  follows  the  fonn  of  a  MMNL  probability  model,  with  utility 
equations  composed  of  alternative-specific  constants  (ASC)  /.-Ls,  and  normally  distributed 
“error  components”  x^-s.  The  nonlinear  equations  above  comprise  the  first-order  condition  for 
the  maximum  likelihood  estimation  (i.e.,  first  derivative  of  the  log-likelihood  function  equated  to 
zero)  for  such  an  MMNL  model.  In  the  job  choice  modeling  context,  the  left-hand-side  is  the 
predicted  share  of  the  Mi  job  in  the  population,  which  is  equated  to  qk,  the  percentage  of 
applicants  that  actually  chose  the  Mi  job.  These  observations  imply  that  we  can  solve  for  the  job 
specific  constants  that  satisfy  the  job  quota  constraints  using  an  MMNL  parameter  estimation 
algorithm.  Biogeme  (Bierle,  2003)  model  files  for  carrying  out  the  estimation  of  selected  optimal 
classification  problems  are  specified  in  Appendix  B. 

While  based  on  a  somewhat  awkward  interpretation  of  criterion  estimates  (a'c--s)  as  “error 
components,”  the  above  observation  provides  a  connection  between  Brogden’s  optimal  job 
classification  and  DCM  from  a  computational  viewpoint.  In  the  next  section,  we  provide  a  more 
useful  conceptual  relationship  between  optimal  classification  and  DCM  that  will  lead  to  a  more 
practical  framework  for  optimal  classification  analysis. 


Calculating  MPP 


As  in  de  Corte’s  method,  the  MPP  can  be  evaluated  by  taking  expectations  using  the  PDF 
of  the  estimated  criterion  score  corresponding  to  the  optimal  job  of  a  randomly  chosen  applicant 
under  the  modified  classification  algorithm.  The  PDF  under  the  modified  classification  algorithm 
is  given  by 


wit 

/*&£»  =  ^  fxk(x) 


expf  A(  A.  +  x 


<=L 


W™"1 27=i  exP  U(A  +  *«)) 


Evaluating  expectations  using  this  PDF,  we  obtain  the  following  expression  for  the  MPP  (see 
Appendix  A): 


MPP 


expUCA  +xifc)} 
x - 

5y=1exp(*(^+xfj) 
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/*(*)  dx 


(4) 


Note  that  MPP  (4)  corresponds  to  (2)  under  the  MVN  model.  The  PDF  /..,-  (x)  involves 

the  unknown  job  specific  constants  as  parameters.  This  PDF  is  completely  specified  by  solving 
the  job  quota  system  of  nonlinear  equations  for  the  unknown  constants. 

Comparison  Between  MMNL  and  MVN  Optimal  Criterion  PDFs 

We  briefly  comment  on  the  relationship  between  the  PDFs  of  the  optimal  criterion 
estimate  derived  by  De  Corte  under  the  original  Brogden  classification  algorithm  and  the 
corresponding  PDF  under  the  modified  classification  algorithm.  Both  PDFs  t,.f,x)  and  f^{x) 

have  the  same  general  form  but  the  terms  inside  their  summation  differ  slightly.  The  integration 
in  the  final  expression  for  /..  *\x)  is  the  conditional  MMNL  probability  of  identifying  the  Mi  job 

as  optimal  with  a  nonnal  mixing  distribution  of  dimension  (m- 1),  given  by  fXi  |A-  (  .r,  .  It 

corresponds  to  the  lower  tail  probability  in  /iw(x),  which  equals  the  probability  of  identifying 

xi 

the  Mi  job  as  optimal  using  the  conditional  probit  model  /  v  |  -  •  1  x,  k  f\x ) .  Apart  from  missing 
the  MNL  probability,  the  integral  in  fxhix)  is  evaluated  only  on  the  subspace 

(*,  +  Aj  <  x  -b  Ak  ij  ^  k  .  This  is  because  under  Brogden’s  original  algorithm,  the  remaining 
(m- 1)  criterion  estimates  are  restricted  once  the  optimal  job  is  identified,  while  under  the 
modified  algorithm  the  (m- 1)  criterion  estimates  can  be  any  point  in  R  ~ 1  since  the  Gumbel 
random  variable  components  can  ultimately  determine  the  optimal  job.  Lastly,  it  is  easy  to  verify 
that 


expl  A(Ak  +x)) 

lllll  - 

so  that  as  becomes  large  the  integration  in  becomes  approximately  equal  to  m(x),  as 

expected. 


)) 


=  i 


ixj+At<x+Ah 


Simultaneous  Selection-Classification  Problem 

For  completeness,  we  derive  the  MMNL  approximation  model  for  evaluating  MPP  under 
simultaneous  optimal  selection-classification  using  our  proposed  algorithm.  In  selection- 
classification,  the  sum  of  the  m  job  quotas  is  less  than  100  percent,  with  the  remainder  equal  to 
the  rejection  percentage.  In  his  multivariate  normal  implementation,  De  Corte  handled  the 
selection-classification  problem  by  introducing  a  cutoff  value  for  the  marginal  PDF  of  .V.  .  This 
cutoff  value  became  an  additional  variable  in  the  system  of  nonlinear  equations  representing  job 
quota  constraints  (see  equation  (9)  in  De  Corte),  along  with  the  m  job  specific  constants.  For  the 
modified  classification  algorithm,  we  expand  the  RUM  interpretation  of  the  optimal  assignment 
rule.  We  accomplish  this  by  including  an  “auxiliary  job”  where  non-accessions  or  rejected 
applicants  are  “assigned.”  Using  index  /  =  0  to  identify  the  non-accession  job,  the  augmented 
criterion  estimate  (or  utility)  corresponding  to  this  non-accession  job  is  given  by 
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where  is  an  additional  job  specific  constant  and  £jc  is  an  extra  standard  Gumbel  random 
variable  that  is  independent  of  the  other  m  Ei;  s.  Behaviorally,  to  identify  the  optimal  job  for  an 
applicant,  we  simply  treat  the  non-accession  just  like  any  of  the  other  m  jobs.  In  this  case,  the 
non-accession  job  is  the  “optimal  job”  for  an  applicant  if 

+  X(J)  +  Ei;-  <  AAq  +  EiC  ;j^0 
Likewise,  we  assign  the  applicant  to  the  Mi  job  if 

ii*0+B„<aCM*+xa,)  +  rw 

and 

KAi  +  xu)  +  EU  <  +  Eij  ij  *  k 

Using  the  above  optimal  assignment  rules,  the  job  quota  constraint  for  the  selection- 
classification  problem  becomes 


expi/Uc3 


expi  re,)  +  SfLi  exp  ( A(Aji  +  xtj)  ) 

_ exp(d(^,.  +  xik)) _ 

exp(AA0)  +  £J=1  exp  ( A(^;  + 


fxixddxi  =  *7c 


fxlxi)dxi  =  qk:  k=  l, 


m 


where  qc  =  1  —  S  ,’=i  q  ■■  Note  that  the  left-hand  side  expressions  are  just  the  probabilities  of 
assigning  an  applicant  to  the  non-accession  job  and  the  Mi  job.  As  in  the  classification-only 
problem,  solving  for  the  (to  —  1  1  job-specific  constants  above  can  be  viewed  as  an  MMNL 
model  parameter  estimation  problem.  Mathematically,  the  constant  qc  is  equivalent  to  the  cut  off 
value  in  De  Corte’s  approach.  It  might  be  worth  investigating  if  other  RUM  models  can  be 
applied  to  solve  other  variations  of  classification  (e.g.,  multi-stage,  etc.) 

Example  utility  equations  and  input  data  requirements  of  an  MMNL  model  for 
approximating  Brogden  optimal  classification  using  the  Biogeme  model  file  syntax  and  data 
format  are  described  in  Appendix  B. 
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Empirical  (Sample  Based)  Optimal  Classification  Analysis 

In  this  section  we  draw  insights  from  analytical  results  derived  in  the  previous  section  to 
propose  a  more  practical  approach  for  conducting  classification  efficiency  analysis.  We  will 
show  that  this  approach  can  readily  accommodate  personnel  classification  policy  constraints  and 
applicant  preferences,  and  provide  a  more  accurate  representation  of  the  “true”  distribution  of  the 
applicant  pool.  As  such,  it  can  produce  more  realistic  results  compared  to  the  “exact”  MVN  and 
MMNL  methods  presented  in  the  previous  section. 

Limitations  of  a  MVN  Population  Classification  Model 

Brogden’s  optimal  classification  method  assumes  that  the  sample  size  approaches  infinity 
and  that  job  criterion  estimates  are  distributed  as  multivariate  nonnal.  In  applications,  MPP 
calculations  are  based  on  a  multivariate  nonnal  population  with  mean  and  covariance  computed 
from  a  large  sample  of  criterion  estimates.  Both  De  Corte’s  MVN  implementation  and  our 
proposed  MMNL  approximation  replaced  the  sample  of  criterion  estimates  from  actual 
applicants  with  a  multivariate  nonnal  population.  In  other  words,  both  implementations  use  a 
smooth  distribution  to  represent  the  true  applicant  distribution. 

Assuming  a  multivariate  nonnal  distribution  for  the  criterion  estimates  made 
classification  analysis  mathematically  tractable.  There  are  two  key  problems  with  this  approach, 
however.  First,  the  true  distribution  of  criterion  estimates  is  usually  not  exactly  normal.  Because 
of  the  nature  of  optimization,  the  analysis  can  be  sensitive  to  departures  from  nonnality, 
especially  as  the  number  of  jobs  increases.  In  Army  applications  with  a  very  large  applicant 
sample,  discarding  the  sample  data  and  instead  using  a  multivariate  nonnal  distribution  to 
approximate  the  distribution  of  applicant  criterion  estimates  is  wasteful,  when  the  sample  of 
criterion  estimates  computed  from  the  data  offers  a  more  accurate  representation  of  the  true 
distribution.2  Classification  analysis  that  directly  uses  the  sample  of  criterion  estimates  will  be 
more  robust  to  departures  from  normality.  Second,  it  is  difficult  to  mathematically  include  real 
world  classification  constraints,  such  as  cut  scores  and  gender  restrictions,  under  the  multivariate 
nonnal  assumption.  Classification  analysis  that  can  accommodate  such  constraints  will  give 
more  realistic  policy  guidance. 

MNL  Model  for  Optimal  Classification 

We  now  obtain  an  approach  for  classification  efficiency  analysis  that  takes  advantage  of 
sample  data  (i.e.,  not  based  on  an  assumed  mathematical  distribution).  We  first  derive  a  model 
that  relaxes  the  multivariate  nonnal  distribution  assumption  for  applicant  criterion  estimates 
without  eligibility  constraints. 

Implied  MNL  Model 

An  empirical  or  sample  data  approach  for  optimal  classification  analysis  is  implied  in  the 
MMNL  approximation  model  presented  in  the  previous  section.  To  derive  this  empirical 


2  Classification  efficiency  is  concerned  with  criterion  estimates  and  not  with  actual  criterion  values. 
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approach  we  reexamine  the  system  of  nonlinear  equations  (3)  used  to  represent  job  quota 
constraints  under  the  MMNL  approximation  model.  These  equations  are  reproduced  below: 

f  exp 

■'**«*"  £T=i  exp  (  k(A,  +  x(j 


)) 


fxAXi)dxi  ^  1,  , HI 


As  mentioned  earlier,  these  equations  are  equivalent  to  the  first-order  conditions  for  the 
Maximum  Likelihood  Estimator  (MLE)  of  an  ASC-only  MMNL  model  with  correlated  “error 
components”  (i.e.,  acCj,s)  that  are  distributed  as  multivariate  normal  and  observed  job  choice 
percentages  given  by  q^-s.  We  used  this  interpretation  to  derive  the  MMNL  maximum  likelihood 
approach  for  computing  the  job  specific  constants  in  the  modified  augmented  criterion  estimate. 
In  the  MMNL  classification  model,  the  population  of  applicant  criterion  estimates  was 
represented  by  the  multivariate  normal  distributed  error  components.  To  satisfy  the  job  quota 
constraints,  a  synthetic  estimation  dataset  was  constructed  with  number  of  observations  equal  to 
number  of  jobs.  Each  observation  in  this  dataset  represents  a  unique  chosen  job,  with  observation 
weight  equal  to  the  corresponding  job  quota. 

When  a  large  sample  of  applicants  is  available,  we  can  replace  the  multivariate  nonnal 
distributed  error  components  in  the  left-hand-side  (LHS)  of  the  job  quota  constraint  equations 
with  the  applicants’  criterion  estimates.  The  original  MMNL  approximating  model  now  becomes 
a  simple  MNL  model  with  only  job-specific  constants.  We  construct  the  needed  synthetic 
estimation  dataset  that  satisfies  the  job  quota  constraints  by:  (a)  replicating  each  applicant 
observation  in  the  sample  as  many  times  as  the  number  of  jobs,  with  each  replicate 
corresponding  to  the  applicant  choosing  a  unique  job;  and  (b)  setting  the  weight  for  each 
observation  replicate  to  be  equal  to  the  quota  for  the  job  chosen  in  the  replicate.  The  second  step 
ensures  that  the  job  proportions  of  job  “choices”  in  the  synthetic  estimation  data  correspond  to 
the  target  job  quotas. 


Replacing  integration  with  respect  to  the  MVN  distribution  with  a  summation  over  the 
sample  observation  replicates,  the  job  quota  constraint  equations  becomes 


ly  exp  (MAk^xik)) 

71  «=i  £7=1  +  *„}) 


(5) 


where  n  is  the  total  sample  size.3  Note  that  for  each  :  the  left-hand  side  is  simply  the  average  of 
the  optimal  assignment  probabilities  of  applicants  in  the  sample  for  the  Mi  job.  The  preceding 
equations  are  the  first-order  conditions  for  an  empirical  MNL  model  with  x^  s  as  alternative- 
specific  predictors  with  fixed  coefficient  (fl  =  1),  As  as  unknown  ASCs,  and  qks  as  observed 


3  For  given  job  k,  the  LHS  involves  the  sum  of  weighted  probabilities  for  the  Ath  job  across  n  *  to  observation 
replicates.  However,  since  the  weights  for  the  m  replicates  of  an  applicant  add  up  to  one,  the  sum  of  weighted 
probabilities  across  observation  replicates  simplifies  to  the  sum  of  probabilities  across  unreplicated  observations  as 
shown  above. 
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job  choice  percentages.  We  can  solve  for  the  -4.s  by  applying  the  maximum  likelihood  estimation 
for  this  MNL  model  using  the  aforementioned  synthetic  estimation  data. 


To  obtain  the  computational  fonnula  for  the  MPP  under  the  empirical  MNL 
approximation  model,  we  examine  the  expression  of  the  MPP  under  the  MMNL  population 
based  approximation  model: 


MPP  =  f  f  v  ex*^--V!>S) 

••*!*«"  £7=1  e*p(*C^  +  *i;)) 


fx(x)  dx 


Again,  replacing  the  integration  with  respect  to  the  multivariate  nonnal  PDF  fx\  xt )  with 
summation  over  the  sample  data,  we  obtain  the  computational  expression  below  for  the  MPP: 


MPP  = 


m  n 


xp{KAk  +  *»)) 


exi 


lk^T=iexp(^AJ  +  x^) 


(6) 


In  other  words,  the  MPP  in  the  MNL  model  is  simply  the  weighted  average  of  criterion  estimates 
across  all  applicants  and  jobs,  with  MNL  optimal  job  assignment  probabilities  as  weights.  Note 
that  the  general  fonn  of  this  fonnula  is  the  same  as  the  formula  for  computing  the  average  bonus 
in  JCM  models  (Diaz,  Ingerick,  &  Sticha,  2007b;  Diaz,  Sticha,  Hogan,  Mackin,  &  Greenston, 
2012).  The  difference  is  that  here  probabilities  are  derived  from  optimal  assignment  rules  while 
the  JCM  probabilities  were  derived  from  actual  applicant  choices. 


Illustration  Using  Simulated  Data 

We  illustrate  the  MNL  model  for  classification  analysis  using  criterion  estimates  that 
were  simulated  from  a  multivariate  nonnal  distribution  with  zero  mean  and  covariance  matrix 


0.5 

0.2 

0.2 

0.2 

0.2 

0.6 

0.2 

0.2 

0.2 

0.2 

0.7 

0.2 

.0.2 

0.2 

0.2 

0.5. 

The  Biogeme  MNL  model  file  and  input  data  specification  are  described  in  Example  4  of 
Appendix  B.  Table  1  shows  the  job  specific  constants  and  job  level  and  overall  MPP  obtained 
under  the  sample  based  MNL  model  for  optimal  classification  analysis.  For  comparison,  we  also 
show  the  results  obtained  for  the  population  based  MVN  and  MMNL  model  for  the  correlated 
criterion  estimates  examples  in  Appendix  B.  The  constants  and  MPP  shown  in  Table  1  for  the 
MVN  population  model  were  computed  using  implementation  of  De  Corte’s  method.  The 
constants  for  the  MMNL  population  model  were  computed  using  Biogeme  with  model  and  data 
files  as  specified  in  Example  2  of  Appendix  B.  (We  did  not  implement  the  procedure  for 
computing  the  MPP  under  the  MMNL  approximation  model.  The  MPPs  under  the  MMNL  model 
should  be  very  close  to  those  computed  using  the  MNL  sample  based  model.)  The  constants  and 
MPP  for  the  empirical  MNL  model  were  computed  as  described  in  this  section. 
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Overall,  calculated  values  for  the  job  specific  constants  closely  match  across  the  three 
methods.  The  job  level  and  overall  MPPs  calculated  under  the  MVN  population  model  and  MNL 
sample  based  model  also  match  very  closely.  These  observations  are  not  surprising  since  the 
sample  data  used  in  the  MNL  model  were  simulated  from  the  same  multivariate  normal 
population  assumed  in  MVN  example.  In  the  next  examples  we  will  compare  the  results  of  the 
empirical  MNL  model  to  results  of  the  population  based  MVN  and  MMNL  models  when 
accounting  for  eligibility  requirements. 


Table  1  .Comparison  of  Sample  Based  MNL  Results  with  Population  Models 


Model 

Jobl 

Job2 

Job3 

Job4 

All 

Job  Specific  Constants 

MNL 

0.2719 

0.1089 

-0.1273 

-0.2758 

MVN 

0.2719 

0.1204 

-0.1298 

-0.2625 

MMNL 

0.2719 

0.0966 

-0.1300 

-0.2730 

Mean  Predicted  Performance 

MNL 

0.3947 

0.5972 

0.8438 

0.7385 

0.5796 

MVN 

0.3996 

0.5940 

0.8484 

0.7489 

0.5827 

MMNL 

Including  Applicant  Eligibility  Constraints 

Current  applications  of  Brogden  classification  efficiency  analysis  ignore  important 
personnel  classification  constraints,  such  as  gender  restrictions  and  cut  scores.  This  limitation  is 
inherent  in  a  method  that  is  mainly  based  on  an  assumed  multivariate  nonnal  distribution.  It  can 
substantially  impact  the  results  of  the  analysis,  depending  on  how  the  true  distribution  of 
criterion  estimates  differs  from  a  multivariate  normal  distribution.  For  example,  gender 
restriction  makes  combat  jobs  off  limit  to  a  sizeable  subset  of  the  applicant  population, 
effectively  reducing  size  of  the  optimization  space.  Therefore,  ignoring  this  restriction  will  likely 
produce  higher  overall  MPP  than  possible  operationally.  In  practice,  this  could  mean  arriving  at 
inaccurate  or  incorrect  conclusions  regarding  the  incremental  classification  efficiency  potential 
of  experimental  predictor  batteries  for  the  full  accession  cohort  or  for  specific  jobs.  In  this 
section,  we  will  generalize  the  sample  data  based  job  quota  and  MPP  equations  (5)  and  (6)  to  the 
case  where  applicant  eligibility  constraints  are  included  in  the  problem. 

Modified  MNL  Model 

Limitations  of  the  Brogden  optimal  classification  approach  related  to  applicant  eligibility 
can  be  easily  fixed  using  the  empirical  MNL  optimal  classification  model.  In  an  MNL  model,  as 
with  all  DCMs,  the  analysis  data  can  include  indicator  variables  identifying  which  alternatives 
were  available  to  individual  decision  makers.  Using  these  indicator  variables,  we  can  readily 
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include  policy  constraints  related  to  applicant  job  eligibility  in  the  MNL  classification  model. 
Making  some  jobs  unavailable  to  some  applicants  requires  adjustments  to  the  weights  of 
observation  replicates  to  ensure  that  the  proportions  of  “chosen”  jobs  are  equal  to  the  targeted 
job  quotas.  It  also  requires  modification  to  the  job  quota  constraint  equations  because  only  those 
applicants  who  are  eligible  for  a  given  job  contribute  to  the  quota  for  the  job. 

The  adjusted  replicate  weights  are  constructed  in  two  steps  as  follows.  First,  we  obtain 
provisional  replicate  weights  by  proportionately  allocating  the  contribution  of  an  applicant  to  job 
quotas  based  on  his  eligibility  profile.  Using  the  indicator  variable  <SC>  to  denote  the  /th 
applicant’s  eligibility  for  the  rth  job,  these  provisional  replicate  weights  are  given  by 

*  SircIr 


Note  that  qi:.  =  0  if  the  applicant  is  not  eligible  for  the  rth  job.  Second,  we  adjust  the  provisional 
weights  so  that  the  sum  of  the  final  replicate  observation  weights  for  the yth  job  is  proportional  to 
the  quota  for  the  job,  yielding  final  replicate  observation  weights  given  by 

w  =  a*  — ^ — 
w  tr  *  cy  -  ' 

Note  that  m-.;.  =  qr  ,  r  =  1,  ...,m  as  desired. 

We  can  now  construct  the  job  quota  constraint  equations  (i.e.,  first-order  MLE 
conditions)  for  the  MNL  optimal  classification  model  with  applicant  job  eligibility  constraints  by 
summing  probabilities  across  the  n  X  m  replicate  observations,  as  follows: 


to 

HI 


w. 

r 


i=l  r=  1 


5ikexpjMAk  +  xikj) 
E"  !  exp  (a(  as  +  *„)) 


=  qk  ,k  =  1,  ....to 


(7) 


The  overall  MPP  is  calculated  under  the  modified  MNL  model  with  eligibility  constraints 
by  simply  taking  the  weighted  average  of  criterion  estimates  across  replicate  observations,  with 
weights  equal  to  the  products  of  the  optimal  job  assignment  probabilities  and  replicate  weights. 
This  weighted  average  is  of  the  fonn 


TO  to  TO, 

v1 


MPP  = 


=  m~1LLL 


xi » 


fe=l c=l r-1 


w. 

wl V 


Sik  exp(  A(j4j.  +  Ta)) 


E?=i6ij  \KAi  +  xj) 


(8) 


The  summation  above  is  simply  the  sum  of  the  product  of  applicant  criterion  estimates  and  their 
corresponding  assignment  probabilities,  using  only  applicants  eligible  for  each  job.  Note  that  the 
equations  (7)  and  (8)  simplify  to  equations  (5)  and  (6)  if  each  applicant  is  eligible  for  all  m  jobs. 
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Examples  Using  Actual  Data 


We  present  illustrative  examples  of  the  sample  based  MNL  optimal  classification  model 
using  data  from  the  Tier  One  Performance  Screen  (TOPS)  initial  operational  test  and  evaluation 
study  (Knapp  &  Heffner  2012).  In  these  examples  we  evaluate  the  classification  potential  of 
ASVAB  and  TAPAS  predictors  for  minimizing  6-month  attrition  in  four  MOS,  under  pure 
classification  and  selection-classification  systems,  while  accounting  for  classification  policy 
requirements.  The  two  classification  requirements  considered  are  MOS  minimum  aptitude  area 
(AA)  score  and  gender  restriction.  Table  2  shows  the  four  MOS,  their  minimum  AA  score 
requirement,  and  gender  restriction. 

The  initial  steps  in  applying  the  MNL  optimal  classification  model  in  this  problem  are  as 
follows.  The  first  step  is  to  estimate  the  probability  of  6-month  attrition,  the  criterion  estimate  of 
interest,  using  data  from  each  of  the  four  MOS.  To  evaluate  incremental  classification  efficiency 
of  TAPAS  beyond  the  ASVAB,  we  estimated  two  logistic  models  for  each  MOS,  one  using  only 
ASVAB  scores  as  predictors  and  the  other  using  both  ASVAB  and  TAPAS  scores  as  predictors. 
The  second  step  is  to  apply  the  estimated  logistic  models  to  a  larger  sample  of  applicants  to 
obtain  an  empirical  distribution  of  criterion  estimates  in  the  applicant  population.  We  used  a 
large  sample  of  accessions  (n=62,155)  from  the  TOPS  study  to  represent  the  applicant  input. 
Table  2  shows  the  sizes  of  MOS  samples  used  to  develop  the  6-month  attrition  logistic 
probability  models,  observed  attrition  in  the  sample,  and  pseudo  R2  of  the  estimated  models 
under  the  two  sets  of  predictors  corrected  for  restriction  in  range  relative  to  the  full  sample.4 
Overall  observed  attrition  for  all  four  MOS  combined  is  1 1.5%. 

Note  that  with  attrition  probabilities  as  criterion  estimates,  the  normal  distribution 
assumption  in  de  Corte’s  approach  no  longer  holds.  To  get  around  this  problem,  Trippe,  Diaz  and 
Ingerick  (2012)  used  the  linear  predictor  function  in  the  logistic  model  to  determine  optimal 
classification  of  applicants.  They  then  evaluated  the  mean  attrition  for  each  MOS  by  taking 
expectation  with  respect  to  the  distribution  of  linear  predictors  of  optimally  classified  applicants 
in  each  MOS.  This  is  a  reasonable  approach  because  the  linear  predictor  is  monotonically  related 
to  the  probability  of  attrition.  Because  the  linear  predictor  is  a  weighted  sum  of  continuous 
variables  (i.e.,  ASVAB  and  TAPAS  scores)  the  normal  distribution  assumption  is  also  tenable. 

Under  the  MNL  optimal  classification  model  the  distribution  of  the  criterion  estimates  is 
detennined  empirically  from  sample  data.  Therefore  it  is  possible  to  directly  use  attrition 
probability  estimates  to  detennine  the  optimal  classification  of  applicants.  In  the  first  set  of 
examples  below,  applicants  are  classified  using  the  linear  predictor  under  four  classification 
conditions  based  on  rejection  rate  and  classification  policy  constraints.  These  conditions  are 
shown  in  Table  3  in  order  of  increasing  number  of  constraints.  In  the  second  set  of  examples 
applicants  are  classified  directly  using  attrition  probability  estimates. 


Correction  for  restriction  in  range  was  carried  out  by  applying  McKelvey  and  Zavoina  (1975)  pseudo  R2  to  the  full 


sample,  R "  =  .  ",  where  -  •  is  the  linear  predictor  of  the  logistic  model  P  tv  =  1 1  = 


e\y'3X> 

L+SXp‘l3X,< 
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Table  2.  Eligibility  Requirements  and  6-Month  Attrition  for  Selected  MOS 


Eligibility  Conditions  _ 6-Month  Attrition _ 

Actual  _ Model  Pseudo  R2 


MOS 

Cut  Score 

Gender 

N 

Attrition 

ASVAB 

AS  VAB+T  APAS 

1  IB 

CO>=87 

Male  Only 

4702 

12.14 

0.0257 

0.0710 

31B 

ST>=91 

NA 

264 

12.12 

0.0851 

0.2130 

68W 

ST>=101 

NA 

990 

8.08 

0.0470 

0.1203 

88M 

OF>=85 

NA 

662 

11.33 

0.0466 

0.1572 

Table  3.  Classification  Conditions 


Reject 

Eligibility 

Description 

0% 

None 

Pure  classification,  ignoring  cut  scores  and  gender  restriction 

10% 

None 

Selection-Classification,  ignoring  cut  scores  and  gender  restriction 

10% 

CS 

Selection-Classification,  enforcing  MOS  cut  scores  only 

10% 

CS  +  (1  IB  Male) 

Selection-Classification,  enforcing  MOS  cut  scores  and  gender 
restriction 

Table  4  shows  the  mean  6-month  average  attrition  estimate  for  the  entire  sample  and  by 
MOS  when  applicants  are  optimally  classified  using  the  linear  predictor  under  different 
conditions  and  predictors,  with  allocation  percentages  of  46.2%,  18.9%,  22.9%,  and  13.0%,  for 
1  IB,  3  IB,  68W,  and  88M,  respectively.  The  table  is  organized  to  facilitate  comparison  of  results 
between  classification  conditions  for  a  given  set  of  predictors.  Compared  to  actual  attrition 
percentages  in  the  sample,  optimally  classifying  applicants  using  only  ASVAB  scores  with  no 
policy  constraints  produced  substantial  improvements  in  mean  attrition  probabilities  for  the 
entire  sample  and  by  MOS  except  for  1  IB.  This  result  is  not  surprising  given  that  1  IB  accounts 
for  close  to  half  of  the  sample  and  has  a  pseudo-R2  that  is  practically  zero. 

Rejecting  10%  of  applicants  and  optimally  classifying  the  remaining  90%  with  no 
constraints  reduced  overall  and  MOS  attrition  probabilities  by  about  half  percentage  point.5 
Adding  a  cut  score  constraint  when  classifying  90%  of  the  applicants  increased  overall  attrition 
from  8.0%  to  8.1%.  While  negligible  this  is  the  anticipated  direction  in  mean  attrition  due  to 
addition  of  the  cut  score  constraint.  Note  that  mean  attrition  for  68W  increase  from  6.0%  to  6.1% 
while  mean  attrition  for  3  IB  increase  from  5.6%  to  5.8%.  Again,  while  negligible,  changes 
observed  for  3  IB  and  68 W  are  meaningful  given  that  both  MOS  use  the  same  AA  composite 
(ST),  with  68W  having  the  higher  cut  score  (ST>=101)  compared  to  3  IB  (ST>=91).  In  other 
words,  3  IB  and  68W  more  directly  compete  with  each  other  than  with  1  IB  and  88M,  with  68W 
having  priority  to  better  applicants  (with  ST  >=101)  compared  to  3  IB.  Including  a  gender 
restriction  in  addition  to  the  cut  score  constraint  produced  an  additional  increase  in  overall 
attrition,  from  8.1%  to  8.2%,  with  1  IB  having  the  largest  increase  (10.5%  to  10.8%)  as  expected. 


5  A  small  rejection  rate  of  10%  was  used  because  3%  of  accessions  do  not  qualify  for  any  of  the  four  MOS  when 
using  both  cut  score  and  gender  eligibility  constraints. 
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Table  4.  Summary  of  Classification  Potential  of  the  AS  VAB  and  TAP  AS  for  Minimizing  6- 
Month  Attrition  Using  Logistic  Linear  Composite  Optimal  Classification  by  Condition 


Reject 

Eligibility 

11B 

31B 

68W 

88M 

ALL 

ASVAB  Only 

0% 

None 

11.1 

6.0 

6.5 

6.6 

8.5 

10% 

None 

10.5 

5.6 

6.1 

6.2 

8.0 

10% 

CS 

10.5 

5.8 

6.0 

6.2 

8.1 

10% 

CS  +  (1  IB  Male) 

10.8 

5.8 

6.1 

6.3 

8.2 

ASVAB+TAPAS 

0% 

None 

12.1 

3.6 

5.3 

3.3 

7.9 

10% 

None 

10.7 

3.2 

4.7 

2.9 

7.0 

10% 

CS 

10.9 

3.3 

4.7 

3.0 

7.1 

10% 

CS  +  (1  IB  Male) 

11.1 

3.4 

4.8 

3.1 

7.3 

Incremental  Efficiency  of  TAPAS  Over  ASVAB 

0% 

None 

-1.0 

2.4 

1.2 

3.3 

0.7 

10% 

None 

-0.2 

2.5 

1.4 

3.2 

1.1 

10% 

CS 

-0.4 

2.5 

1.3 

3.3 

1.0 

10% 

CS  +  (1  IB  Male) 

-0.4 

2.5 

1.3 

3.2 

1.0 

The  second  set  of  results  in  Table  4  summarizes  the  potential  classification  benefits  of 
using  both  TAPAS  and  ASVAB  scores  on  6-month  attrition.  Optimally  classifying  applicants 
using  TAPAS  and  ASVAB  scores  yielded  substantial  reduction  in  average  attrition  over  the 
classification  model  based  only  on  ASVAB  for  the  entire  sample  and  by  MOS  with  exception  of 
1  IB.  The  reduction  in  estimated  attrition  rate  is  substantial  (about  50%)  for  MOS  3  IB  and  88M 
across  all  conditions.  MOS  68W  showed  a  modest  decrease  in  estimated  attrition.  The  estimated 
attrition  for  MOS  1  IB  unexpectedly  went  up  by  less  than  one  percentage  point  across  all 
conditions.  The  general  pattern  of  small  increases  in  attrition  rates  as  cut  score  and  gender 
restriction  constraints  are  included  in  the  classification  model  is  the  same  as  those  observed  in 
the  model  using  only  ASVAB. 

Table  5  shows  the  mean  6-month  attrition  probabilities  for  the  entire  sample  and  by  MOS 
when  applicants  are  optimally  classified  directly  using  attrition  probability  estimates  based  on 
the  logistic  model.  The  first  set  of  results  in  Table  5  shows  the  potential  classification  benefits  of 
using  only  ASVAB  scores  as  predictors  under  different  conditions.  Using  the  logistic  probability 
estimates  of  attrition  to  optimally  classify  applicants  produced  approximately  the  same  estimated 
attrition  rates  for  the  overall  sample  across  conditions,  compared  to  classification  based  on  the 
linear  predictor.  The  estimated  attrition  rates  by  MOS  when  classifying  using  attrition  probability 
estimates  are  also  generally  comparable  to  the  attrition  rates  when  classification  is  based  on  the 
linear  predictor  (see  Table  4).  However,  the  attrition  rates  for  MOS  3  IB,  68W,  and  88M  increase 
by  about  a  half  percentage  point,  while  the  attrition  rates  for  MOS  1  IB  decrease  by  0.6  to  0.9 
percentage  points. 
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Table  5.  Summary  of  Classification  Potential  of  the  AS  VAB  and 
TAPAS  for  Minimizing  6  Month  Attrition  Using  Logistic 
Probability  Optimal  Classification  by  Condition 


Reject 

Eligibility 

11B 

31B 

68W 

88M 

ALL 

ASVAB  Only 

0% 

None 

10.2 

6.5 

7.3 

6.9 

8.4 

10% 

None 

9.9 

6.0 

6.7 

6.4 

8.0 

10% 

CS 

9.9 

6.1 

6.5 

6.6 

8.0 

10% 

CS  +  (1  IB  Male) 

10.1 

6.2 

6.7 

6.7 

8.2 

ASVAB+TAPAS 
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10% 

CS  +  (1  IB  Male) 
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0.9 
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1.3 

10% 

CS 

0.8 

1.9 

1.2 

2.2 

1.2 

10% 

CS  +  (1  IB  Male) 

0.9 

1.8 

1.1 

2.2 

1.3 

The  second  set  of  results  in  Table  5  summarizes  the  potential  classification  benefits  of 
using  both  TAPAS  and  ASVAB  scores  on  6-month  attrition  when  applicants  are  classified 
directly  using  attrition  probability  estimates.  Compared  to  classifying  applicants  using  the  linear 
predictor,  the  overall  attrition  estimates  are  lower  by  about  0.4  percentage  points  when 
classifying  applicants  directly  using  the  estimated  attrition  probabilities.  At  the  MOS  level, 
attrition  rates  for  MOS  3  IB,  68W,  and  88M  increased,  approximately  1  to  2  percentage  points, 
while  the  attrition  rates  for  MOS  1  IB  decreased  by  1.7  to  2.4  percentage  points,  when  classifying 
applicants  using  attrition  probability  estimates  compared  to  classifying  using  linear  predictor. 

Comparing  the  results  for  the  two  sets  of  predictors  in  Table  5  shows  that  optimally 
classifying  applicants  using  TAPAS  and  ASVAB  scores  yielded  substantial  reduction  in  average 
6-month  attrition  for  the  entire  sample  and  all  MOS  over  the  classification  model  based  only  on 
ASVAB.  Note  that  unlike  classification  using  the  linear  predictor,  classifying  applicants  using 
the  estimated  attrition  probability  shows  that  MOS  1  IB  also  benefits  from  the  inclusion  of 
TAPAS  in  addition  to  the  ASVAB,  along  with  a  higher  incremental  improvement  for  the  entire 
sample.  In  other  words,  classifying  applicants  directly  using  attrition  probability  estimates 
produced  stronger  evidence  of  classification  efficiency  compared  to  classification  using  the 
linear  predictor. 

Using  a  linear  predictor  appeared  to  have  favored  MOS  with  lower  estimated  attrition 
probabilities  (or  higher  probabilities  of  completing  the  first  term)  at  the  expense  of  MOS  1  IB. 
This  “bias”  is  likely  due  to  the  fact  that  a  linear  predictor  is  no  longer  linearly  proportional  to 
logistic  probability  estimates  for  completing  the  term  in  the  upper  tail  of  the  distribution.  For  the 
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same  differences  in  linear  predictors  in  the  tails  and  middle  portion  of  the  logistic  model,  the 
corresponding  differences  in  probabilities  between  jobs  are  small.  Therefore,  classifying 
applicants  using  a  linear  predictor  would  tend  to  prioritize  better  applicants  to  jobs  with  lower 
attrition  estimates  even  if  they  can  contribute  better  to  lowering  overall  attrition  in  jobs  with 
higher  attrition.  Figure  1  graphically  shows  the  bias  of  classification  using  a  linear  predictor 
(solid  lines)  over  direct  classification  using  attrition  probability  estimates  (dashed  lines).  Direct 
classification  shifted  the  distribution  of  attrition  probability  estimates  for  MOS  3  IB,  68W,  and 
88M  to  the  right,  with  a  corresponding  shift  to  the  left  in  the  distribution  for  MOS  1  IB.  The  net 
effect  of  these  shifts  is  higher  overall  classification  efficiency,  as  noted  in  observations  from 
Tables  4  and  5. 

In  sum,  we  applied  the  MNL  optimal  classification  model  to  evaluate  the  potential 
classification  efficiency  of  ASVAB  and  TAPAS  when  classification  policy  constraints  are 
included  in  the  model.  While  the  differences  in  average  attrition  estimates  as  policy  constraints 
were  included  in  the  classification  model  were  negligible,  the  pattern  of  changes  were  consistent 
with  anticipated  effects  of  the  cut  score  and  gender  restriction  on  the  MOS  in  the  example.  We 
expect  these  effects  to  be  stronger  with  more  valid  criterion  models  and  greater  differentiation  in 
classification  policy.  For  example,  cut  score  policy  differentiation  between  MOS  3  IB  (ST>=91) 
and  35T  (ST>=1 12)  is  greater  compared  to  that  between  MOS  3  IB  and  68W  (ST>=91).  We  also 
demonstrated  that,  unlike  de  Corte’s  implementation  of  Brogden  classification,  the  empirical 
MNL  optimal  classification  model  is  not  dependent  on  approximately  normally  distributed 
criterion  estimates.  Classifying  applicants  directly  using  estimated  attrition  probabilities  yielded 
more  accurate  and  stronger  evidence  of  incremental  classification  benefits  of  TAPAS  over 
ASVAB  for  minimizing  6-month  attrition. 
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Figure  1.  Comparison  of  6-Month  Attrition  Probability  Distributions  Under  Optimal 
Classifications  Using  Linear  Predictor  and  Logistic  Probability  with  10%  Rejection  and 
Cut  Score  and  Gender  Restriction 
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Hybrid  Optimal  Classification-Choice  Model 


We  briefly  describe  a  hybrid  optimal  classification  model  that  includes  applicant 
preferences.  Using  a  utility  interpretation,  we  can  further  modify  the  augmented  criterion 
estimate  to  allow  applicant  preferences  to  affect  the  optimal  classification  solution.  For  example, 
letting  2)  represent  the  incentive  for  the  /th  job  that  the  /th  applicant  is  eligible  for,  the 
augmented  criterion  estimate  can  be  modified  as  follows  so  that  classification  optimization  is 
conditional  on  applicant  preferences  for  the  incentives: 

XW),i  =  K*,  +  *<;)  + 

Ill  the  equation  above, 

B  is  a  fixed  parameter  in  the  MNL  model,  with  unknown  constants  Us  that  will  be  computed  to 
satisfy  the  job  quota  constraints  for  given  scale  parameter.  Note  that  the  additional  term  BZt, 
dilutes  the  effect  of  the  original  criterion  estimate,  Xt  i ,  in  identifying  the  job  that  will  provide  the 
greatest  contribution  to  the  average  predicted  criterion  estimate  of  applicants.  Instead  of 
maximizing  the  criterion  estimates  A'.  ,s  subject  only  to  job  quota  constraints,  the  modified 
classification  rule  would  classify  an  applicant  to  a  job  with  lower  predicted  criterion  estimate  if 
the  applicant’s  preference  for  the  incentive  offered  is  relatively  higher. 

We  outline  below  a  general  approach  for  carrying  out  this  hybrid  model  for  solving  an 
optimal  classification  problem  conditional  on  applicant  preferences  for  incentives.  First,  using 
data  with  actual  applicant  job  choices  and  incentives  A.  ,s,  estimate  a  job  choice  model  with 
utility  equations 

U„=A‘j+A'Xl]4-B‘Zll  +  Ell 


In  the  utility  equation  above, the  term  A‘W; 
represents  the  “observed  classification  effect”  in  the  data.  In  previous  studies,  we  found 
statistically  significant  classification  effects  for  AA  scores  of  applicants  (Diaz,  Ingerick,  & 
Sticha,  2007a;  Diaz,  Ingerick,  &  Sticha,  2007b;  Diaz,  Sticha,  Hogan,  Mackin,  &  Greenston, 
2012).  Unlike  in  the  classification  problem,  the  constants  /I ,  s  represent  average  preferences  not 
explained  by  the  incentive  and  classification  composite.  Next,  expand  the  modified  augmented 
criterion  estimate  as 

W)«  ■  *(*1 + x‘i + BZ>i) + Eu 

with  B  =  Bi  /A 

,  where  B  is  the  coefficient  of  the  incentive  estimated  from  the  choice  model.  This  expanded 
augmented  criterion  estimate  is  equivalent  to  the  scaled  utility  equation  for  the  MNL  model, 
where  5  is  known  and  fixed  and -Us  are  job  specific  constants  that  satisfy  the  quota  constraints. 
As  before,  these  job  specific  constants  can  be  solved  using  standard  MNL  estimation  procedures. 
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Conceptually,  the  coefficient  B  in  the  expanded  augmented  criterion  estimate  represents 
the  dilutive  effect  of  incentives  in  the  hybrid  classification  model.  Setting  the  scale  constant  to  ' 
will  induce  an  optimal  classification  model  in  which  the  effect  of  applicant  preferences  for 
incentives  relative  to  criterion  estimates  is  equal  to  that  observed  in  the  data.  This  classification 
model  will  in  fact  be  equal  to  the  choice  model  except  that  the  job  specific  constants  are 
detennined  to  satisfy  target  job  quota  constraints  instead  of  the  observed  job  choice  proportions. 
We  can  use  the  scale  constant  to  impose  realistic  limits  to  potential  classification  benefits  or 
effectiveness  of  composites  or  predictors.  Scale  values  larger  than  '  presume  classification 
effects  that  are  greater  than  observed.  Setting  the  scale  constant  to  extremely  large  enough  values 
would  effectively  eliminate  the  effects  of  incentives  in  the  classification.  From  a  practical  view, 
the  important  point  is  that  competing  policies,  such  as  incentive  programs,  can  potentially  limit 
the  classification  benefits  of  alternative  predictors  for  maximizing  average  perfonnance.  The 
scale  constant  provides  a  mechanism  for  imposing  this  limit  if  desired.  Conversely,  the  hybrid 
model  can  also  inform  how  alternative  incentive  policies  affect  average  predicted  performance  of 
applicants. 


Discussion  and  Recommendations 

In  this  research  we  showed  how  the  Brogden  classification  framework  can  be  fonnulated 
using  a  DCM  framework.  We  first  specified  a  MMNL  classification  model  that  is  analytically 
comparable  to  the  MVN-based  solution  proposed  by  de  Corte.  To  obtain  the  MMNL 
classification  model,  we  rescaled  the  augmented  criterion  estimates  proposed  by  Brogden  using 
an  arbitrarily  large  constant  and  then  added  independent  standard  Gumbel  errors.  This 
transformation  produced  modified  augmented  criterion  estimates  with  the  same  structure  as  the 
utility  equations  in  a  MMNL  model  with  error  components  corresponding  to  the  original 
criterion  estimates.  Taking  advantage  of  the  similarity  in  interpreting  job  specific  constants,  we 
showed  analytically  and  by  numerical  examples  that  the  constants  that  satisfy  the  job  quota 
constraints  in  Brogden’ s  classification  problem  can  be  obtained  using  the  standard  MMNL 
model  estimation  method.  We  also  showed  how  to  specify  a  MMNL  selection-classification 
model  by  interpreting  the  modified  augmented  criterion  estimates  as  utility  equations  and 
applying  random  utility  maximization. 

We  also  proposed  an  empirical  or  sample  based  MNL  classification  model  for  evaluating 
potential  classification  efficiency  that  accommodates  personnel  classification  policy  constraints 
and  is  robust  with  regard  to  the  fonn  and  distribution  of  criterion  estimates.  For  classification 
problems  where  a  large  amount  of  applicant  data  is  available,  the  empirical  distribution  of 
criterion  estimates  is  a  more  accurate  representation  of  the  true  distribution  in  the  applicant 
population.  The  MNL  classification  model  produced  classification  probabilities  by  fractionally 
weighting  individual  applicants  to  different  jobs  to  obtain  the  highest  overall  predicted  criterion. 

We  illustrated  the  sample  data  based  MNL  optimal  classification  model  by  evaluating  the 
classification  efficiency  potential  of  ASVAB  and  TAPAS  for  minimizing  6-month  attrition  in 
four  MOS  under  pure  classification  and  selection-classification  models  and  varying  classification 
policy  requirements  using  data  from  the  TOPS  study.  While  changes  in  estimated  attrition  due  to 
inclusion  of  cut  scores  and  gender  restriction  were  negligible,  their  directions  were  consistent 
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with  anticipated  effects  of  the  policy  constraints.  We  expect  these  effects  to  be  stronger  with  a 
more  valid  criterion  model  and  greater  differentiation  in  eligibility  constraints  among  jobs.  The 
6-month  attrition  example  showed  that  the  choice  of  classification  composites  can  substantially 
affect  the  results  of  optimal  classification  analysis.  Using  the  linear  predictor  in  the  logistic 
model  as  a  proxy  for  attrition  probability  estimates  produced  somewhat  lower  overall 
classification  efficiency  compared  to  direct  classification  using  attrition  probability  estimates, 
and  inaccurately  indicated  that  TAPAS  does  not  provide  incremental  classification  benefit  for 
MOS  1  IB  beyond  the  ASVAB.  The  same  inaccurate  indication  is  expected  from  the  MVN 
distribution  based  analytic  approach  of  de  Corte,  using  a  linear  predictor  as  the  best  MVN 
distributed  proxy  for  the  attrition  probability  estimate. 

The  constant  /I  used  to  scale  the  modified  criterion  estimates  can  have  varying  theoretical 
and  practical  implications  in  applications  of  the  MNL  classification  model.  On  one  extreme,  as 
the  scale  approaches  zero,  the  Gumbel  error  tenn  will  dominate  the  modified  augmented 
criterion  estimate,  producing  classification  that  would  be  close  to  random.  On  the  other  extreme, 
as  the  constant  becomes  large,  the  elements  in  the  optimal  classification  probability  matrix  in  the 
MNL  classification  model  would  generally  approach  ones  or  zeroes,  assuming  criterion  estimates 
are  continuous  variables.  This  probability  matrix  of  ones  and  zeroes  is  comparable  to  the 
decision  matrix  in  a  binary  integer  linear  programming  (BILP)  model  for  matching  applicants  to 
jobs  to  maximize  the  average  predicted  criterion.  In  other  words,  the  empirical  MNL 
classification  model  converges  to  the  BILP  model  as  the  scale  constant  increases  to  infinity.  For 
values  not  large  enough  to  produce  classification  probabilities  of  ones  or  zeroes,  the  scale 
constant  has  a  practical  interpretation  as  a  parameter  of  “classification  uncertainty.”  This 
uncertainty  may  be  viewed  as  unreliability  or  measurement  errors  of  criterion  and  predictor 
variables.  It  can  also  be  related  to  shrinkage  corrections  to  obtain  cross-validated  prediction 
results.  These  interpretations  can  be  used  to  specify  a  scale  constant  that  will  produce  the  desired 
correction  due  to  unreliability  and/or  shrinkage  when  evaluating  the  potential  classification 
efficiency  of  predictors. 

We  also  outlined  a  hybrid  classification-choice  model  for  evaluating  the  classification 
potential  of  predictors  that  accounts  for  applicant  preferences.  In  real  world  classification, 
applicant  preferences  for  incentives  can  potentially  limit  the  classification  benefits  of  alternative 
predictors.  The  hybrid  approach  provides  a  mechanism  for  imposing  this  limit  if  desired. 
Conversely,  the  hybrid  model  can  also  inform  how  alternative  incentive  policies  affect  average 
predicted  performance  of  applicants. 

In  sum,  the  DCM  provided  an  alternative  framework  for  carrying  out  Brogden’s  optimal 
classification  of  applicants  to  jobs.  The  mathematical  population  based  MMNL  model  specified 
in  this  research  provides  a  computationally  more  convenient  alternative  to  the  MVN  based 
method  for  evaluating  potential  classification  efficiency.  The  sample  data  based  MNL  model  is 
more  robust  with  respect  to  distributional  assumptions  about  the  criterion  estimates  and  can 
easily  accommodate  policy  constraints,  thereby  producing  more  accurate  results  with  respect  to 
potential  classification  efficiency  of  alternative  predictors.  Lastly,  concepts  underlying  the  DCM 
based  framework  for  classification  can  be  applied  to  develop  a  hybrid  classification-choice 
model  that  can  be  used  to  add  realism  to  optimal  classification  analysis  (e.g.,  by  considering  the 
effects  of  enlistment  incentives  and  applicant  job  preferences). 
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Appendix  A:  Derivations  Under  Modified  Augmented  Criterion 


Job  Quota  Constraints  Equation 

We  begin  by  specifying  flh '  k) i,  the  probability  that  the  Ath  job  of  a  randomly  selected 

applicant  has  the  highest  modified  augmented  criterion  under  the  modified  optimal  classification 
algorithm.  Using  X(a'^s  for  classifying  applicants,  the  optimal  job  is  determined  jointly  by 
Xt  =  (Xn, ,  AC.,, )  and  =  (Eilr ... ,  Ei;v  ).  We  use  conditional  probability  in  our  derivation 
below  to  break  the  joint  probability  of  (Xit  Et)  into  manageable  parts. 


Conditioning  on  YE  —  xi  and  integrating  over  all  possible  values  ,  the  probability  that 
the  Ath  job  has  the  highest  modified  augmented  criterion  estimate  is  obtained  as  follows: 

/;*(»  =  P(MAJ  +  Xa)  +  EH  <  KA;  +  xik)  +  B*  ij*k) 

=  fx.eRv  P{KAj  +  xu)  +  Bit  K  +  xik)  +  Bat\xi  =  *iiJ* 

=  Kk?  =  *|*t  =  xi)fxi(xi)dxi 


Substituting  the  MNL  conditional  p rob ability.*  expression 
for  P[K^  =  k\Xi  =  x. ),  we  obtain  the  system  of  nonlinear  equations  below,  with  variables  -As, 
representing  job  quota  constraints  under  the  modified  classification  algorithm: 


exp(A(A,+3cefe)) 
E7=i  exp  (A(Af  + 


fxixjdxi  =  qk  ,k  =  1,  ...,m 


MPP  Equation 

We  derive  .*..  •(  the  PDF  of  the  estimated  criterion  score  corresponding  to  the  optimal 

job  of  a  randomly  chosen  applicant  under  the  modified  classification  algorithm.  To  begin,  we 
note  that  conditional  on  observing  the  m  criterion  estimates, 

=*,)  =  s?=1p(A-f  ~  xkPf = 

Note  that  P{K*  =  k  ) 

is  the  MNL  probability  defined  earlier  and  l(arfc=®)  is  an  indicator  function  that  evaluates  to  one 
if  Ad.  =  x  and  zero  otherwise.  The  first  line  above  simply  means  that  the  PDF  of  .V:  given 
ATj  =  xt  is  the  probability  that  the  optimal  predicted  criterion  score  can  be  anyone  of  the  m 
observed  predicted  criterion  scores.  The  indicator  function  1^=., ,  ensures  that  /..A  v|Afj  =  x(  j 
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evaluates  to  non-zero  probabilities  only  at  the  observed  criterion  estimates  (i.e.,  The 

second  line  is  obtained  by  noting  that  Xf  =  x}  implies  K[‘  =  & . 

Using  the  above  conditional  probability  representation,  we  derived  below  the 
unconditional  PDF  of  X* .  To  facilitate  the  derivation  below,  we  partition  the  vector  of  criterion 
estimates  into  the  scalar  Xk,  representing  the  criterion  estimate  for  the  Mi  job,  and  the  (m- 1 ) 
vector  X(k ,,  representing  all  other  criterion  estimates.  Again,  by  conditioning  on  Yj  =  xt  and 
integrating  over  all  possible  values  Jq,  we  can  derive  -‘..xj  as  follows: 


=  |  ^  =  xi)  fxi  (xddxi 


' Xi€R ‘ 
m 


-  T  I  P(K[l  —  —  xi)  l(xk=se})fx^  xk)  fx&'ilx*  £.*(*) 

hi J*^ 

m 

=  2  /  m-i  P'"A^  =  k  l*W  =  X(k>Xk  =  X)fxk 


i-=  i  -V* 


fe“l 

m 


fc-1 


e*pU(A  +  x'j) 
,*i**m_1  Zf=1  exp(^(x^  +  x£j) 


Again, this  PDF  involves  the  unknown  job  specific  constants  as  parameters.  To  completely 
specify  f,M.xj  we  need  to  solve  the  job  quota  system  of  nonlinear  equations  for  -As. 

After  completely  specifying  f,.:-X_x)  we  can  evaluate  the  MPP  under  the  modified  optimal 
classification  algorithm  using 


MPP  =  £  Jj*4«  j  f  ^x^dx^ 

^  41  'xieRm~±  1  exp  (A(A  +  xti)) 


dx 


,  -*«*-*»»  f 


-Yf  . 

L7=1«q>U(^+*,y)) 
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Appendix  B:  Biogeme  MMNL  Model  Approximation  Examples 

The  following  examples  describe  the  utility  equations  and  input  data  requirements  of  an 
MMNL  model  for  approximating  Brogden  optimal  classification  using  the  Biogeme  model  file 
syntax  and  data  fonnat.  In  these  examples  we  will  evaluate  the  classification  efficiency  of  a 
predictor  battery  for  four  jobs.  We  will  assume  that  the  predicted  criterion  estimates  have  been 
standardized  and  that  we  have  available  the  covariance  (correlation)  across  the  four  jobs.  The 
three  examples  below  demonstrate  the  MMNL  approximation  for  Brogden  optimal  classification 
depending  on  the  structure  of  the  predictor  covariance/correlation  and  type  of  problem,  whether 
classification-only  or  simultaneous  selection-classification. 

Example  1:  Classification  with  Uncorrelated  Predictors 

This  example  shows  how  to  specify  an  MMNL  model  and  input  data  to  approximate 
Brogden’ s  optimal  classification  when  predictors  are  uncorrelated  and  the  accession  rate  is  100 
percent  (i.e.,  pure  classification  problem).  Lor  this  problem,  assume  that  the  predicted  criteria  are 
uncorrelated  across  jobs  with  squared-validities  equal  to  0.50,  0.60,  0.70,  and  0.50  and  the  target 
job  allocations  are  40%,  30%,  20%,  and  10%. 

The  top  part  in  Ligure  1  shows  sections  of  the  Biogeme  model  file  specifying  an  MMNL 
model  for  approximating  an  optimal  classification  problem  with  four  jobs  and  uncorrelated 
predictors.  The  section  labeled  [Utilities]  specifies  the  utility  equations  for  the  four  jobs 
(alternatives),  with  each  row  representing  one  job.  The  first  three  columns  provide,  respectively, 
unique  ID  numbers  for  identifying  the  “chosen  alternative”  in  the  input  data  file,  unique 
alphanumeric  labels  for  identifying  alternatives  in  Biogeme’s  output,  and  the  name  of  the 
variable  in  the  input  data  file  that  indicates  “availability”  of  each  alternative.  The  last  column 
(shown  in  bold  text)  describes  the  utility  equations  using  a  parametcrx variable  syntax.  Using  this 
syntax,  the  term  “  ”  represents  the  job-specific  constants  for  the  first  job,  where 

is  the  (fixed)  job  specific  constant  parameter  and  “  ”  is  a  variable  whose  constant  value 

(equals  1)  is  specified  under  the  [Expressions]  section  of  the  model  file.  The  criterion  estimates 
are  represented  by  the  error  component  terms  using  the  syntax  “ 

where  the  string  “  ”  represents  a  random  parameter  with  mean  and 

standard  deviation 
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The  section  labeled  [Beta]  describes  how  parameters  are  handled  during  estimation.  The 
first  column  identifies  the  parameters  in  the  utility  equations.  The  next  three  columns  specify 
starting  values  and  bounds  for  the  parameters,  while  the  last  column  indicate  whether  the  value 
of  a  parameter  will  be  fixed  at  the  specified  starting  value  (status=l)  or  will  be  estimated  in  the 
problem  (status=0).  For  approximating  Brogden’s  optimal  classification,  the  only  parameters  in 
the  MMNL  model  to  be  estimated  are  the  job  specific  constants  (i.e.,  ASC).6  The  means  and 
variances  of  the  criterion  estimates  (error  components  in  the  utility  equation)  are  usually 
computed  separately  from  analysis  data.  For  this  example,  the  predictors  have  means  all  equal  to 
zero  and  standard  deviation  0.707107,  0.774597,  0.836660,  and  0.707107  (i.e.,  validities 
corresponding  to  R~  values  of  0.50,  0.60,  0.70,  and  0.50). 

The  last  two  sections  in  the  model  file,  [Group]  and  [Scale],  are  used  to  specify  the 
arbitrary  value  of  the  scaling  parameter  .In  nonnal  applications  of  Biogeme,  these  two  sections 
are  used  in  combination  to  specify  a  scale  heterogeneous  model  (i.e.,  with  different  scales  across 
groups  of  observations  or  applicants).  For  our  particular  problem,  we  use  these  sections  to 
specify  an  arbitrary  scale  for  the  entire  data,  as  follows.  The  expression  ( 

)  under  the  [Group]  section  evaluates  to  1  for  all  four  observations/jobs  (see  following 
discussion),  effectively  specifying  a  single  scale  for  all  four  jobs.  This  scale  is  specified  as  a 
fixed  parameter  under  the  [Scale]  section.  For  this  example  we  are  using  '  =  100. 

The  bottom  part  in  Figure  1  shows  the  desired  format  of  the  input  data  for  the  MMNL 
model  approximation  in  this  example.  The  first  row  identifies  the  variables  corresponding  to  the 
columns  in  the  data.  The  data  shown  in  Figure  1  is  in  weighted  form,  with  each  row  observation 
representing  a  portion  of  the  estimation  data.  Instead  of  showing  individual  observations  (as  in  a 
JCM  estimation  problem),  this  data  format  describes  the  distribution  of  applicants  who  “chose” 
(or  will  be  assigned  to)  each  job.  The  “chosen”  job  represented  in  each  row  is  identified  under 
the  column,  while  the  percentage  of  applicants  who  “chose”  each  job  is  given  under  the 

column.  To  obtain  estimated  ASCs  that  satisfy  the  job  quota  constraints,  we  specify 
weights  that  are  equal  or  proportional  to  the  quota  percentages  of  the  job  corresponding  to  each 
row.  In  this  example,  the  job  quota  constraints  are  40%,  30%,  20%,  and  10%,  respectively,  for 
jobs  1,  2,  3,  and  4.  Therefore,  we  created  a  choice  data  with  applicant  job  choice  percentages 
equal  to  these  values. 


Example  2:  Classification  with  Correlated  Predictors 

This  example  shows  how  to  specify  an  MMNL  model  and  input  data  to  approximate 
Brogden’s  optimal  classification  when  predictors  are  correlated  and  the  accession  rate  is  100 
percent  (i.e.,  pure  classification  problem).  We  will  use  the  same  set  of  job  quota  constraints, 
40%,  30%,  20%,  and  10%,  as  in  the  first  example,  but  specify  the  job  predictor  covariance 


6  Note  that  normalization  involves  fixing  the  value  of  one  of  the  ASCs.  In  this  example  we  fixed  the  ASC  for  the 
first  job  at  the  final  estimated  value  obtained  using  our  implementation  of  De  Corte’s  method.  Our  implementation 
of  De  Corte’s  method  did  not  set  the  constant  for  the  first  job  to  zero,  as  he  recommended.  In  order  to  make  direct 
comparison  between  the  constants  obtained  under  the  two  methods,  we  fixed  the  ASC  in  the  approximating  MMNL 
model  to  that  obtained  using  the  multivariate  normal  approach. 
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Figure  2  shows  the  required  Biogeme  MMNL  model  file  when  criterion  estimates  are 
correlated.  Because  we  are  using  the  same  job  quota  constraints  as  in  the  first  example,  the 
required  input  data  will  be  the  same  as  that  shown  in  Figure  1 .  The  utility  equations  specified  in 
Figure  2  involve  job-specific  constants  and  random  error  components  with  longer  expressions, 
which  are  based  on  the  Cholesky  decomposition  of  the  covariance  matrix  V  =  LLT  where  L  is  a 
lower  triangular  matrix  with  positive  diagonal  entries.  This  factorization  is  often  used  to 
construct  or  generate  a  random  vector  with  specified  covariance  from  independently  distributed 
random  variables.  For  example,  if  Z  =  (Zx,  ...,Z,„  )*  is  a  random  vector  with  elements  that  are 
independently  distributed  as  standard  normal,  then  F  =  LZ  is  a  random  vector  that  is  distributed 
as  multivariate  nonnal  with  zero  mean  and  variance  V. 


In  Figure  2  the  error  components  in  the  utility  equations  are  just  the  rows  in  the  4- 
dimensional  vector  F  obtained  below: 
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The  values  of  the  elements  of  the  lower  triangular  matrix  L  ■ .  s)  are  specified  under  the  section 
[Expressions]  as  variables  with  fixed  values  across  observations.  The  standard  nonnal  random 
variables  2]s  are  specified  using  Biogeme’s  random  parameter  syntax  (e.g.,  Zl  =  FmuO  [  Fsdl  ], 
Z2  =  FmuO  [  Fsd2  ],  etc). 
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Example  3:  Selection-Classification  with  Correlated  Predictors 


This  example  shows  how  to  specify  an  MMNL  model  and  input  data  to  approximate 
Brogden’s  optimal  classification  when  predictors  are  correlated  and  the  accession  rate  is  less 
than  100  percent  (i.e.,  selection-classification  problem).  For  this  example  we  will  use  a  rejection 
rate  of  30%  and  distribute  the  remaining  70%  using  the  percentages  in  the  first  two  examples. 
This  allocation  produces  job  quota  constraints  of  28%,  21%,  14%,  and  7%  for  the  four  jobs  and 
30%  for  the  dummy  job  for  non-accession.  We  also  specify  criterion  predictor  correlations  using 
the  covariance  matrix  given  in  the  second  example. 

Figure  3  shows  the  [Beta]  and  [Utilities]  sections  in  the  Biogeme  model  file  and  the  input 
data  needed  for  solving  the  selection-classification  problem  using  an  MMNL  model.  In 
specifying  the  Biogeme  model  file,  we  simply  modified  the  model  file  used  in  the  second 
example  by  adding  the  utility  equation  for  the  non-accession  job.  The  input  data  is  also  modified 
by  adding  a  fifth  observation  with  weight  equal  to  the  rejection  rate  and  setting  the  weights  for 
the  other  four  observations  to  28%,  21%,  14%,  and  7%.  All  other  components  of  the  model  file 
are  the  same  as  in  Figure  2. 
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Figure  B-3.  Selected  Sections  of  the  Model  File  and  Input  Data  for  Approximating  a 
Selection-Classification  Problem  with  Correlated  Predictors 


Selected  Sections  of  the  Model  File: 


Input  Data: 


Example  4:  Empirical  (Sample  Based)  MNL  Classification  Model 

Figure  4  shows  relevant  sections  of  the  Biogeme  model  file  and  the  structure  of  the  input 
data  for  carrying  out  the  MNL  classification  model  analysis  using  criterion  estimates  that  were 
simulated  from  a  multivariate  nonnal  distribution  with  zero  mean  and  the  same  covariance 
matrix  as  in  the  Example  2.  There  are  two  main  changes  in  Figure  4  compared  to  the  MMNL 
model  files  in  Example  2.  First,  individual  applicant  criterion  estimates  observations  (XI,  X2, 
X3,  X4)  are  directly  included  in  the  model,  instead  of  being  represented  as  error  components. 
Second,  actual  sample  applicant  observations  comprised  the  input  data,  with  each  observation 
replicated  as  many  times  as  the  number  of  jobs  using  job  quota  percentages  as  replicate  weights. 
Technically,  the  sample  observations  are  used  as  “grid  points”  for  the  empirical  distribution 
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represented  by  the  sample  data.7  Note  that  correlation  of  criterion  estimates  are  implicitly  taken 
care  of  by  the  sample  data.  As  before,  we  use  a  large  value  for  the  scale  parameter  (■'  =  100)  to 
ensure  that  differences  between  criterion  estimates  drive  the  assignment  algorithm. 


7  Under  the  simulated  MLE  algorithm  for  the  MMNL  approximation  method,  the  grid  points  correspond  to  the 
random  values  generated  to  approximate  the  multivariate  normal  distribution  of  the  error  components/criterion 
estimates. 
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